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Abstract 

One of the central challenges facing modern neuroscience is to explain the ability of the nervous 
system to coherently integrate information across distinct functional modules in the absence of 
a central executive. To this end Tononi et al. [Proc. Nat. Acad. Sci. USA 91, 5033 (1994)] 
proposed a measure of neural complexity that purports to capture this property based on mutual 
information between complementary subsets of a system. Neural complexity, so defined, is one of a 
family of information theoretic metrics developed to measure the balance between the segregation 
and integration of a system's dynamics. One key question arising for such measures involves 
understanding how they are influenced by network topology. Sporns et al. [Cereb. Cortex 10, 127 
(2000)] employed numerical models in order to determine the dependence of neural complexity on 
the topological features of a network. However, a complete picture has yet to be established. While 
De Lucia et al. [Phys. Rev. E 71, 016114 (2005)] made the first attempts at an analytical account 
of this relationship, their work utilized a formulation of neural complexity that, we argue, did not 
reflect the intuitions of the original work. 

In this paper we start by describing weighted connection matrices formed by applying a random 
continuous weight distribution to binary adjacency matrices. This allows us to derive an approx- 
imation for neural complexity in terms of the moments of the weight distribution and elementary 
graph motifs. In particular we explicitly establish a dependency of neural complexity on cyclic 
graph motifs. 

PACS numbers: 87.18. Sn, 87.19.1o, 87.19.1j, 89.75.Fb, 89.70. Cf, 02. 10. Ox 
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I. INTRODUCTION 



Graph theory has been employed with some success within neuroscience [1]. However 
graph theory explicitly concerns itself with characterizing network structure and at best can 
deliver only circumstantial insight into network dynamics. As a partial answer to this, over 
the past decade neuroscientists have developed a suite of dynamical measures that employ 
information theory [2-4]. While application of information theoretic measures come with 
their own limiting assumptions (e.g. they typically demand that the dynamics on a network 
be stationary) their ability to deal naturally with weighted connectivity matrices has made 
them extremely appealing. However in general the analytical understanding of such measures 
is less well-developed than the tools of graph theory. Consequently, it is likely that a more 
comprehensive picture of network function could be achieved by an understanding of the 
relationship between information theoretic and graph theoretic measures [5]. In this paper 
we pursue one such analysis. 

Neural complexity, as formulated by Tononi, Sporns and Edeleman [2], is perhaps one 
of the most widely discussed information theoretic measures. It purports to capture the 
nervous system's ability to coherently integrate information at the level of the organism 
while maintaining the functional segregation associated with specialization across a range 
of anatomical scales [6] . 

The TSE measure calculates the mutual information shared between processes occurring 
on complementary subsystems of a neural network in order to identify networks that si- 
multaneously support relatively segregated local behavior and relatively integrated global 
dynamics. Tononi et al. were able to demonstrate that network structures inspired by the 
properties of the cerebral cortex scored highly [2, 7]. Subsequently, the measure has been 
used extensively, e.g. to characterize the dynamics of different regions in the mammalian 
brain [8], to analyze evolved robot controllers [9] and to explore theories of sleep, conscious- 
ness and schizophrenia [10]. 

The relationship between neural complexity and graph topology has been central to the 
intuitions that underlie the neural complexity measure. Originally Sporns et al. [8] found 
that constructing the graph topology of networks using organizational principles derived 
from observations of the cerebral cortex led to high neural complexity. Sporns et al. [7] 
used graph theoretic tools to analyze the topology of networks that were optimized for high 
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complexity using an evolutionary algorithm. De Lucia et al. [11] were the first to attempt to 
decompose neural complexity in terms of elementary graph motifs. They achieved this by 
utilizing a popular analytic model that proceeds on the assumption that neural dynamics 
may be approximated by a stationary multivariate stochastic process and, furthermore, 
that this process is Gaussian. This enables the interactions between network components 
(and mutual information itself) to be expressed via a covariance matrix. Recently we have 
highlighted and then amended an error in this analytical model by moving to a continuous 
time analogue of the original apparently discrete time formulation [12]. Consequently it 
is necessary to revisit the analytical work of De Lucia et al. in light of this correction. 
Furthermore, we argue that the formulation of neural complexity employed by De Lucia et 
al. did not implement correctly the original definition of neural complexity [12]. 

We start by deriving an approximation for neural complexity in terms of elementary 
graph motifs. We find that to a first approximation complexity is dependent on reciprocal 
connections and to a second approximation on two types of 3-cycle. We go on to suggest 
that higher order terms in our approximation for neural complexity will be dependent, with 
diminishing impact, on cycles of increasing order. 

We illustrate the validity of our approximation by example of a ring lattice connectivity 
scheme where the probability of connection between elements decays with the distance be- 
tween them. This scheme may be considered a discretized version of the model discussed 
in detail in [12], which was inspired by a Toeplitz covariance matrix model suggested in [2]. 
The ring lattice model ratifies our application of the neural complexity approximation to 
topological graphs and also reveals conditions for the presence of a peak in complexity at 
intermediate connectivity decay values. 

II. NEURAL COMPLEXITY 

The scenario we address here is the same as in [12] (see also [2, 7, 13, 14]): we have a 
system of n "neural components" (nodes for brevity) and a stationary multivariate stochastic 
process X(t) = {Xj(t)| i = l,...,n} running on the system, where Xi(t) represents the 
activation state at time t of the i th node. In [2] the authors introduced a neural complexity 
measure based on mutual information between subsystems of the given system. The idea 
behind the measure is that complex neural systems should be expected to exhibit a balance 
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between "integration" and "segregation" of neural subsystems. The measure is defined as 
follows: firstly the integration associated with the system is introduced as 

n 

1=1 

where H denotes the entropy H(X(t)) of the full process X(i) and Hi the entropy H(Xi(t)) 
of the individual activation Xi(t). Note that by stationarity these quantities and hence I 
itself do not depend on time t. I may be interpreted as a measure of the deviation from 
independence of the individual components of the system. Neural complexity is then defined 
to be 

71— 1 / , \ 
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where (-) k denotes an average over all subsystems of size k of the given system. Neural 
complexity is thus an average over all scales (represented by subsystem size k) of the differ- 
ence between mean integration (I), at the given scale and the appropriately scaled global 
integration -I. For a highly segregated system both of these quantities will be small and Cj^ 
itself thus small. Conversely, for a highly integrated system the integration for individual 
subsystems will be close to the scaled global integration, and Cm will again be small. C^ 
can thus be expected to attain peak values for systems which are neither highly integrated 
nor highly segregated; see [2] for a fuller discussion of interpretation of the measure. C^ 
may also be expressed directly in terms of entropies as 

n— 1 / , \ 

/ h \ 

(3) 



Actualization of the measure 



In the special case where the X(t) are jointly multivariate Gaussian, the entropy H may 



be expressed simply in terms of the n x n covariance matrix Vt = X(t) T X(t), where the 
over-bar represents an average over the statistical ensemble [15]. By stationarity Q does not 
depend on time t. We then have H = ~ In ([27re] n so that 

n - 1 f k \ 

(4) 
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Tononi et al. [2] consider an n x n connectivity matrix C, where CV,- is to be interpreted 
as the weight on the connection from node i (efferent) to node j (afferent), and a linear 



autoregressive neural process 

X(t) = X(t) ■ C + e(t) (5) 

driven by serially uncorrelated Gaussian noise e(t). However, there is an error in their calcu- 
lation of the covariance matrix associated with the process (5) [12]. While the error is readily 
corrected, in [12] it is argued that such a discrete time process is likely to be unacceptably 
unrealistic and leads, furthermore, to conclusions which probably do not support the intu- 
itions of the originators of the measure [2]. Thus in [12] the discrete time model (5) is 
dropped in favor of the continuous time multivariate Ornstein-Uhlenbeck process [16, 17 

dX(t) = -X(i) • (I - C) dt + dW(t) , (6) 

where W(i) is a multivariate Wiener process with identity covariance matrix, representing 
white noise applied independently to each node [30]. In [12] it is shown that neural com- 
plexity for this process has distinctly different characteristics from that for the discrete time 
process (5) (see [12] for more discussion on this topic). 

Eq. (6) may be viewed as a linearized, noisy Continuous Time Recurrent Neural Network 
(CTRNN) [18]. The condition for existence of a stationary process (6) is 

max {mc(X) | A e Eig(C) } < 1 , (7) 

where Eig(C) denotes the set of eigenvalues of C. The stationary process (6) is multivari- 
ate Gaussian so that (4) applies, with covariance matrix Q satisfying the continuous-time 
Lyapunov equation 

2ft = I + C 1 n + ttC , (8) 

(see [12] for a derivation) for which there exist efficient algorithms for numerical solution 
[19]. 



B. The neural complexity approximation 

From (8) the matrix series expansion 
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may be derived. Defining the order parameter e = ||C|| for some (submultiplicative) matrix 
norm ||-|| [20], in [12] this expansion is the basis for derivation of the approximation 

C N = C* N + C* N *+0(e A ) , (10) 



where 



is 0(e 2 ) and 



C *n = + (11) 



Cn — — gg — ^ ] (3CijCjkCik + CijCjkCki) 



[12) 



is 0(e 3 ). 

In [12] it is recommended that, in order to establish a level playing field when comparing 
complexity between networks, some form of normalization be applied to the connection 
matrix C; in particular spectral normalization, where C is premultiplied by w/p(C) with 

p{C) = max {|A| | A G Eig(C) } (13) 

the spectral radius of C and < w < 1 a scale parameter. Since spectral radius is the 
infimum of all (induced) matrix norms [20] it is a good indicator of the accuracy of the neural 
complexity approximation (10); i.e. the approximation can be expected to be accurate for 
small w. This is borne out empirically [12]. 



III. RELATING NETWORK CONNECTIVITY TO GRAPH STRUCTURE 

The motivation for this paper is to investigate how neural complexity C_\f relates to 
the graph structure of a putative network underlying a neural system. Towards this end 
we require some plausible scheme by which to relate a connectivity matrix C to a given 
adjacency matrix A = (A^j) representing the topology of the underlying network. 

De Lucia et al. [11] introduce a simplified scheme where the connection matrix C is just the 
adjacency matrix, which they take to be symmetric and with no self-connections, normalized 
by system size. They then proceed to derive an approximation to neural complexity C/v for 
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a stationary multivariate Gaussian process on such a network. However, besides taking as 
their starting point the discrete time process (5) [31] which, as mentioned previously, we 
consider unsuited to neural complexity analysis, we believe their analysis to be flawed in 
the following respect: in averaging over /c-subsystems as required by (3), the authors of 
[11] appear effectively to treat subsystems in isolation — i.e. as neural processes in their own 
right, uncoupled from the full system [32] — which surely defeats the purpose of Cj^ as defined 
by (2). This has the result of introducing spurious 2 nd order terms (the correct result for 
the discrete system time system is of 4 th order in connectivity; see [12], eq. 40). 

In this study we consider only directed graphs. Our approach, however, extends straight- 
forwardly to undirected (bi-directional) graphs. Graph topology is specified by a binary 
adjacency matrix Aij, so that Ay = 1 represents a directed connection from node % (effer- 
ent) to node j (afferent). As regards diagonal elements, we take the view here that, while 
our graphs will not have self-connections — so that An = for all % — diagonal elements of 
the connection matrix are to be regarded as representing variation in the characteristic re- 
laxation time, or activation decay, of the individual neural components. See [12] for more 
discussion on this point. In the spirit of statistical physics we now suppose that, given an 
adjacency matrix A, weights are assigned independently and identically at random to each 
connection. We suppose that diagonal activation decay elements are drawn, also identically 
and independently, from a separate distribution with zero mean, so that 



WijAij % ^ j 

(14) 

Di i = j 



where the Wij are iid as some random variable W and the Di iid as some random variable D 
which has mean zero. We may then, given a graph structure A, consider neural complexity 
Cj^f — via the randomness [33] introduced by W, D — as a random variable Cj^{A\ W, D) and 
we define the complexity measure for a graph A to be the expectation over W, D: 

C^(A)=V(C M (A;W,D)) . (15) 

We define similarly the approximations C* N {A), C*^(A). Note that these measures depend 
on the particular distributions W, D used in the construction of the random connectivity 
matrix. As regards normalization, in the case where graphs of similar mean degree are 
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m2,i 11*2,2 m2,3 ni2,4 

FIG. 1: 2-edge motifs: only shaded motifs are implicated in the neural complexity measure. 

compared one might take the view that normalization is less critical, since in some sense the 

overall connectivity strength averaged over all network weights W, D will not vary drastically 

(but see the example in Section IV below). 

Without normalization — and assuming that we may safely ignore instances of C for which 

the stationarity condition (7) fails — we may calculate from (11) and (12) that 

n 4- 1 

C* N (A) = _t-( M2 m 1+ 2 M 2 m 2>2 ) (16) 
n 4- 1 

Cn(A) = ^-/i 3 (m3,3 + m 3 , 8 ) , (17) 

where \i = E(W) and /i 2 = E(jy 2 ) are respectively the mean and second moment of the 
weight distribution, and 

m a = J^Atj (18) 
m 2 , 2 ee iJ^AjAji (19) 
m 3 , 3 ee ^AijAjkAik (20) 

The m Pi g may be interpreted as the multiplicities of certain graph motifs — small repeated 
subgraph fragments, [21], in the graph represented by A. Specifically, mi is just the total 
number of edges of the graph, m 2j2 counts the number of reciprocal connections, while 1x13,3 
and 1x13,8 count the numbers of two varieties of 3-cycle; see FIGS. 1 and 2. Note that 
since activation decay D has zero mean neither C* N (A) nor C^(A) depends on D; there 
may potentially, however, be dependencies on higher order moments of D for higher-order 
approximations. 




Tn.3,9 rr^io tri3,n ^3,12 

FIG. 2: 3-edge motifs: only shaded motifs are implicated in the neural complexity measure. 

Equations (16) and (17) tell us the following: the expected neural complexity of a neural 
system based on a graph according to the prescription (14) depends, to a first approximation, 
on the total number of connections and the number of reciprocal connections in the graph. 
To the next order of approximation it depends on the number of two varieties of 3-cycle. The 
procedure via which C* N and C^f were derived (see [12]) suggests that if we were to calculate 
higher order approximations according to the expansion (9) for the covariance matrix, then 
higher order cyclic graph motifs would successively come into effect with diminishing impact, 
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to a degree depending roughly on the spectral radius of A and the moments of W (and 
possibly of D). 

We may well wish to assess neural complexity for statistical ensembles of graph structure; 
that is, the adjacency matrix elements Ay represent (jointly distributed) random variables. 
In this case, Ctf(A) and its approximations, as well as the motif counts 11X1,1x12,2,11x3,3,1x138 
may themselves be considered as random variables and their ensemble means (and higher 
moments) calculated. In the following we shall use the notation (■) to denote an average 
over a statistical ensemble of graphs, as distinct from the mean E(-) over the network weight 
random variables W, D. 

IV. EXAMPLE 

To illustrate the integration/segregation balance that motivates their formulation of the 
measure Cv, [2] demonstrate a complexity peak for an ad hoc Toeplitz covariance matrix 
where covariance decays as we move away from the diagonal. In [12] this idea is analyzed in 
more detail via our approximation formula. In that paper a connection matrix is constructed 
based on a ring lattice, where connectivity (and consequently covariance) decays exponen- 
tially with lattice distance (see also [22]). The resultant covariance matrix is a Toeplitz 
matrix analogous to that in [2]. Here we introduce a model which parallels the ring lattice 
construction in [12]. Rather than, as in that model, having connection strength decay with 
lattice distance, we instead construct an ensemble of random graphs where the probability of 
connection decays with lattice distance. This model is thus amenable to a graph theoretic 
analysis as expounded above. 

We proceed by constructing a directed random graph with no self-connections on a ring 
lattice of n nodes, where each edge is assigned independently with probability 



for i 7^ j, where d(i,j) = min (|z — j\ , n — \i — j\) is ring lattice distance, < a < 1 a 
connectivity decay parameter [34] and c a constant to be determined. We shall generally 
work to the large network limit n — > 00. We may calculate that in this limit the mean (in or 
out) degree is given by k = 2cr / (1 — a). We shall take n, k and a as the defining parameters 




(22) 
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of the model, so we set 

c=\K{l/a-l) (23) 
in (22). Note that < 1 for all i ^ j requires 

a> a = min(l - 2/«, 0) . (24) 

The probability that two arbitrarily chosen distinct nodes are connected is given by p = n/n; 
it corresponds to the connection probability for an Erdos-Renyi random graph [23] of the 
same size. In the large network limit, the degree distribution of our model tends to a Poisson 
distribution, as for an Erdos-Renyi random graph [23]. Note that while the mean degree of 
individual graphs instantiated from the ensemble may vary, its ensemble average is always 

K. 

We first calculate the mean approximate neural complexity without normalization; to 
this end the chief advantage of our model lies in the mutual independence of the Aij which 
allows us to calculate easily the means of the quantities mi, m 2i 2, rn 3i3 , m 3j8 over the ensemble. 
In the large network limit we find 

(mi) = UK (25) 
<m 2 , 2 ) = (26) 

Ks> = 3 (m 3 , 8 ) = f^ 3 ^ 1 ^, (27) 

valid for a < a < 1. Recall that in (16) and (17) C* N (A) and C*^{A) have already been 
averaged over the weight distribution W applied to a given graph specified by A. For 
fixed n,K,a we may now, using (25) - (27), average again over the ensemble of random 
graphs A(n, n,a) to derive the mean neural complexity approximations (C* N (n, k, a)) and 
(C* N *(n,n,a)). We find 

, \ \ n(n + 1) ( i 9 9 1 — a\ 
{C K (n, K, a)) = 48 I K m + 2 K f \^ J < 28 > 

{Cff{n , K ,a)) = ^.Yf^ . (29) 

For fixed n, k the first term in (C* N (n, n,a)) is constant, while the second term is monotone 
decreasing with a. (C^(n, k, a)) has a maximum at a = |; rewiring initially boosts and then 
degrades the number of 3-cycles. To examine the behavior of the unnormalized approxima- 
tion in more detail, we may calculate that, as a function of a, (C* N {n, k, a)) + (C^(n, a)) 

12 



has an extreme value at 

* 3k/i - 2 



(30) 



9/^ + 2 ' 

although the extremal only represents a maximum (i.e. a complexity peak) if \x < 0, which 
implies a predominance of inhibitory connections. We note that while this condition may 
exclude the mammalian cortex, where the majority of connections are thought to be ex- 
citatory, it may be relevant to early sensory relays which are dominated by inhibition e.g. 
the olfactory bulb or the first stages of the visual system [24]. In any case, there exists the 
possibility of a complexity peak for negative /j; then the condition < a* < 1 implies 

2 

1 < k < 3 and // < (31) 

or 

2 (k- 1\ 2 

« > 3 and -— < ^ < -— 32 

3k \k — 3/ 3k 

(bearing in mind that if /x is too large and negative then the stability condition (7) is likely to 
be violated). Outside of this regime (C* N (n, k, a) + Cp(n, k, a)) is always monotone decreas- 
ing with a: the complexity contribution of reciprocal connections overwhelms, on average, 
the contribution of 3-cycles and there is no complexity peak at intermediate connectivity 
decay. We remark that for a corresponding undirected graph model every link is effectively 
reciprocal and there is then a possibility of a complexity peak, albeit for sparsely-connected 
networks, specifically for mean degree k < 3. 

We now examine neural complexity under spectral normalization. Since this case ap- 
pears intractable to analysis, we carried out simulations as follows: for a series of mean 
degree values k ranging from 2.5 to 5.5 and for a sequence of decay parameters a in the 
range (do, 1) we generated 10 5 random directed graphs of n = 30 nodes, according to the 
prescription of (22). For each graph A, connectivity coefficients C were generated accord- 
ing to (14). There was no decay variance; i.e. the diagonal weights Di were set to zero. 
Inter-node weights Wij were drawn from a binormal distribution including both positive 
(excitatory) and a smaller fraction of negative (inhibitory) weights; specifically, each weight 
was drawn independently with 80% probability from Af(0.5, 0.01) and with 20% probability 
from A/"(— 0.4, 0.01). Spectral normalization was then applied to each resulting connectivity 
matrix with scale parameter w = 0.2 (we verified that at this normalization level + 
was generally a close approximation to the exact value Cj^f). Finally, the means (C* N (n, k, a)) 
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FIG. 3: Mean approximate neural complexity (C* N (n, n,a)) + (Cp(n,K,a)) under spectral 

normalization, plotted against connectivity decay a in 30-node networks for a range of 
mean degree values k. The cut-off at the left of the plots marks the minimum connectivity 

decay ao = 1 — 2/n. See main text for details. 

and {C}f(n, k, a)) were calculated in sample (given the large sample size standard errors for 
the estimated mean were small, although the variance was substantial). 

Results are illustrated in FIG. 3. We see that within the illustrated mean degree range 
there is a connectivity peak which disappears at approximately k > 5. Thus, as long as 
connectivity is not too dense, we can expect a connectivity peak at intermediate connectivity 
decay comparable with that seen in the continuous exponential connectivity decay model 
of [12] under spectral normalization. As in that study, normalization was (at least for 
positive mean connection strength /i) a prerequisite for the appearance of a connectivity 
peak, although in the current model no self-activation was necessary. 

V. DISCUSSION 

The central claim of this study is that neural complexity is driven by the presence of 
specific topological features of a network, namely cyclic motifs, with lower-order cycles 
exerting the greatest influence. While here we carry the topological analysis up to third 
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order in connectivity, our analytical approach points to the emergence of higher-order cycles 
with diminishing influence on neural complexity [12]. 

The original measure, defined for an arbitrary (stationary) stochastic process, is com- 
monly characterized in terms of a multivariate process running on a weighted network. 
Consequently, in addition to specifying a network topology, one must specify weights over 
the network's edges before the measure can be applied. Here we chose a statistical mechanics 
approach where the weights on a graph are drawn from a random distribution. This allows 
us to consider the neural complexity associated with a particular topology by determining 
the mean over the ensemble of weighted networks with which it is associated. 

It is instructive to contrast the behavior of the neural complexity measure for the 
ring-lattice model with exponential connectivity decay described in [12], where connection 
strength decayed with lattice distance, and the behavior of the same measure for the model 
explored here in which connection probability decays with lattice distance. The original 
model demonstrated that without normalization neural complexity decreases monotonically 
with the decay parameter, whereas with normalization there may be an intermediate com- 
plexity peak provided that there is some variance in the time constants of individual nodes. 
In the current model, the influence of the decay parameter on neural complexity is seen 
explicitly to derive from its impact on the relative prevalence of reciprocal connections and 
3-cycles. While the impact of reciprocal connections simply decreases monotonically with 
the decay parameter, that of 3-cycles peaks at an intermediate value. How this balance 
plays out is determined by a network's mean degree and mean connection strength. 

In general, for there to be an intermediate peak in complexity, mean degree must be low in 
order to prevent reciprocal connections from dominating. In addition, for directed graphs, 
the mean connection strength must be negative, implying a predominance of inhibitory 
connections. We have already mentioned that this constraint may only be met by some 
kinds of neural system [24]. For undirected graphs we note that the number of reciprocal 
connections is by definition constant, but low mean degree is still required in order to see 
an intermediate peak in complexity. In contrast to our previous lattice model with decaying 
connection strength [12], here it was not necessary to introduce variability in network node 
time constants. 

We have seen that connectivity normalization can have a significant impact on the be- 
havior of the neural complexity measure. Spectral normalization, for instance, changes the 



15 



regime within which an intermediate complexity peak may appear, specifically accommo- 
dating higher mean degree. 

In summary, while the founding intuition that complexity derives from a balance between 
integration and segregation might lead one to expect that an intermediate complexity peak 
should be a robust property of any model in which a "segregated" lattice-like graph is relaxed 
towards a more "integrated" random graph, we find that the picture is more subtle and that 
relative frequencies of specific cyclic topological features play a crucial role. 

Finally, we note that neural complexity may be considered as just one of a family of 
information theoretic metrics based on the intuition of segregation-integration balance; we 
expect that the methods employed in this paper can be readily extended to related measures. 
Particularly promising are measures such as causal density [4, 25, 26] which, in contrast to 
the "static" mutual information underlying Cj^f, take into account directed information flow 
between network subsystems, expressible in terms of transfer entropy [27, 28]. In future 
studies we intend to extend the analytic techniques presented here to such measures. 
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